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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 . 1 36(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 1 33). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )£3 Responsive to communication(s) filed on 16 January 2003 . 
2a)D This action is FINAL. 2b)(EI This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 
Disposition of Claims 

4) S Claim(s) 1-42 is/are pending in the application. 

4a) Of the above claim(s) 22-26 is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) |EI Claim(s) 1-21 and 27-42 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10) 0 The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

1 1) D The proposed drawing correction filed on is: a)D approved b)D disapproved by the Examiner. 

If approved, corrected drawings are required in reply to this Office action. 

12) D The oath or declaration is objected to by the Examiner. 
Priority under 35 U.S.C. §§ 119 and 120 

13) D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a)H All b)D Some*c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2.D Certified copies of the priority documents have been received in Application No. . 



3.D Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

14) Q Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application). 

a) □ The translation of the foreign language provisional application has been received. 

15) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121. 
Attachment(s) 

1) £3 Notice of References Cited (PTO-892) 4) □ Interview Summary (PTO-413) Paper No(s). . 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) 5) Q Notice of Informal Patent Application (PTO-152) 

3) S Information Disclosure Statement(s) (PTO-1449) Paper No(s) . 6) O Other: 
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DETAILED ACTION 



Election/Restrictions 



1 . Claims 22-26 are withdrawn from further consideration pursuant to 37 CFR 1 . 1 42(b) as 
being drawn to a nonelected invention, there being no allowable generic or linking claim. 
Election was made without traverse in Paper No. 7. 



2. The information disclosure statement filed February 4, 2000 fails to comply with 37 CFR 
1.98(a)(2), which requires a legible copy of each U.S. and foreign patent; each publication or that 
portion which caused it to be listed; and all other information or that portion which caused it to 
be listed. It has been placed in the application file, but the information referred to therein has not 
been considered. 

Please provide a legible copy of the publication by Martin A. Fischler et al., "Random 
Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and 
Automated Cartography," SRI International. Note that the copy provided is missing text in right 
columns and at the bottom pages. 



3. The drawings are objected to under 37 CFR 1.83(a). The drawings must show every 
feature of the invention specified in the claims. Therefore, the "reducing the size of the input 
image" in claims 5, 12, and 19 must be shown or the feature(s) canceled from the claim(s). 
Claim 30 claims of having an "image acquisition apparatus" which "is mounted on other 
apparatus," this feature is as well not shown in the drawing for complete understanding of the 



Information Disclosure Statement 



Drawings 
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Art Unit: 2621 

claim. Claim 42, claims the shape of the markers being a regular polygon, however, this is not 
clear from the drawings. No new matter should be entered. 

A proposed drawing correction or corrected drawings are required in reply to the Office 
action to avoid abandonment of the application. The objection to the drawings will not be held 
in abeyance. 



4. Applicant is reminded of the proper language and format for an abstract of the disclosure. 

The abstract should be in narrative form and generally limited to a single paragraph on a separate 
sheet within the range of 50 to 1 50 words. It is important that the abstract not exceed 1 50 words 
in length since the space provided for the abstract on the computer tape used by the printer is 



5. Claim 39 recites the limitation "the two-dimensional" in line 14 page 94 of claim 39. 
There is insufficient antecedent basis for this limitation in the claim. 



6. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 
A person shall be entitled to a patent unless - 

(e) the invention was described in a patent granted on an application for patent by another filed in the United 
States before the invention thereof by the applicant for patent, or on an international application by another who 
has fulfilled the requirements of paragraphs (1), (2), and (4) of section 371(c) of this title before the invention 
thereof by the applicant for patent. 



Specification 



limited. 



Claim Rejections - 35 USC § 112 



Claim Rejections - 35 USC § 102 



The changes made to 35 U.S.C. 102(e) by the American Inventors Protection Act of 1999 
(AIPA) do not apply to the examination of this application as the application being examined 
was not (1) filed on or after November 29, 2000, or (2) voluntarily published under 35 U.S.C. 
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122(b). Therefore, this application is examined under 35 U.S.C. 102(e) prior to the amendment 
by the AIPA (pre-AIPA 35 U.S.C. 102(e)). 

7. Claims 1-4, 8-1 1, 15-18, 27, and 40 are rejected under 35 U.S.C. 102(e) as being 
anticipated by Leis et al. (USPN 5,828,770). 

With regards to claim 1, which is representative of claims 8, 15, and 27, Leis et al. (Leis) 
discloses a three-dimensional (3D) position and orientation sensing apparatus (Fig. 1) 
comprising: image input means for inputting an image acquired by an image acquisition 
apparatus (CCD sensors 161eft and 16 right, Fig. 1) and having at least three markers (14a-14d 
on object 12, Fig. 1), 3D positional information of which with respect to an object to be measures 
is known in advance (column 3 lines 31-36); region extracting means for extracting a region 
corresponding to each marker on the image (column 3 lines 36-40, column 7 lines 1-67 
emphasizes how each markers are extracted and identified); marker identifying means for 
identifying the individual markers from the characteristics of the appearance of the markers in 
extracted regions (Leis discloses marker identifying mode starting at column 4 line 37 and Leis 
goes in the detail thought column 5 about identifying the markers); and position and orientation 
calculating means for calculating the 3D position and orientation of the object to be measured 
with respect to the image acquisition apparatus, by using positions of the identified markers on 
the image, and the 3D positional information of the markers with respect to the object to be 
measured (column 9 lines 15-20 and shown in steps in Figs. 10A and 10B). Note, as mentioned 
in claim 27, Leis discloses identifying the individual markers by calculating geometric 
characteristic of the region at column 8 lines 32-38. 
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With regards to claim 2, which is representative of claims 9 and 16, Leis discloses the 
recited features of image input means, region extracting means and marker identifying means as 
mentioned in claim 1 above and further having at least four markers (14a-14d show in Fig. 1) 
and marker selecting means for selecting three markers from the markers identified by the 
marker identifying means (column 3 lines 39-41); parameter calculating means for calculating a 
plurality of sets parameters for calculating 3D position and orientation of the object by utilizing 
positions on the image of the three markers selected by marker selecting means and 3D 
positional information of each marker (column 5 lines 49-55, also see the calculating the 
parameters for the each markers describes at column 6 lines 6-10, lines 38-42, etc.); parameter 
selecting means for selecting one set of parameters by evaluating application of parameters 
calculated by the parameter calculating means, to markers not selected by the marker selected 
means (column 4 lines 15-26, the table show the "not selected markers" by the indication of 
"MISSING". Also note that when only one marker has been selected (i.e., 14a), the other marker 
(i.e., 14B) is shown as a not selected marker at column 6 lines 26-29). 

With regards to claim 4, which is representative of claims 11 and 18, Leis discloses the 
recited features of image input means, region extracting means and marker identifying means, 
position and orientation calculating means as mentioned in claim 1 above and further disclosing 
a distance estimating means for estimating a distance to each marker based on the size of the 
identified marker in the image at column 3 lines 64-67, column 41ines 1-3. Note that knowing 
the x-y-z coordinates system and knowing the distance between the two CCD sensors distance 
can be calculated to each marker from the acquisition device. 
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Regards to claim 3, which is representative of claims 10 and 17, Leis et al. discloses a 
parameter improving means for improving one set of parameters selected by parameter selecting 
means (feature recited in claim 2 above), by evaluating the set of parameters through application 
thereto of 3D positional information of each marker with respect to the object to be measured at 
column 6 lines 25-33. The "new" markers shown in Figs. 6-7 are an improvement from "old" 
markers. 

With regards to claim 40, Leis discloses the image acquisition apparatus acquires a 
plurality of images and utilizes the plurality of images at column 10 lines 44-52. 

Claim Rejections - 35 USC § 103 

8. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

9. Claims 5-7, 12-14, 19-21, 29, 31, 33-34, 38-39, and 42 are rejected under 35 U.S.C. 
103(a) as being unpatentable over Leis et al. in combination with Maggioni et al. (USPN 
5,751,843). 

With regards to claim 5, which is representative of claims 12 and 19, Leis et al. (Leis) 
discloses the recited features of image input means, region extracting means and position and 
orientation calculating means as mentioned in claim 1 paragraph 7above. Leis does not 
expressly disclose image reducing means. Maggioni et al. (Maggioni) discloses image reducing 
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means at column 9 lines 30-39. One of ordinary skill in the art would have been motivated to 
reduce the size of the input image to increase the computing speed and to reduce the use of 
memory space. 

Regards to claim 7, which is representative of claims 14 and 21, Maggioni discloses 
feature extracting means for extracting features of the object to be measured (column 4 lines 46- 
54) and means for calibrating the position and orientation of the calculated object (column 4 
lines 3-14). Note, Maggioni discloses a method for detecting the spatial position and orientation 
of marked objects in an image where the marking are extracted and calibrated as discussed at 
column 4. 

With regards to claim 6, which is representative of claims 13 and 20, the recited features 
are the same as those in claims 7, 14 and 21, and the arguments above as to the relevance of Leis 
and Maggioni are incorporated herein. 

Regards to claim 29, Maggioni discloses the means for identifying the individual markers 
by calculating geometric characteristics of the region of each marker within the image measures 
the size of the region of each marker within the image at column 6 lines 47-50, and the means for 
estimating the relative 3D position and orientation of the object with respect to the image 
apparatus calculates an initial estimate of the distance between the image apparatus and marker 
based on the measured value of the size marker region at column 6 lines 50-55. 

With regards to claim 31, Maggioni discloses estimating value by calculating only a 
marker that takes a range, which makes it possible to estimate the relative distance at column 6 
lines 50-58. Maggioni estimates the relative distance by knowing the x- and y-direction of the 
marking structure. 
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Regarding claims 33 and 34, Maggioni discloses markers wherein the markers are 
circular markers (column 6 lines 17-20) a projected image of each marker within the image is 
approximated to an ellipse (column 6 lines 65-67 and column 7 lines 1-2), and the distance from 
the image acquisition apparatus to each marker is estimated by utilizing the length of the long 
axis of the ellipse (column 7 lines 5-60). Maggioni teaches obtaining the distance from the 
image apparatus to each marker by estimating the long axis of the ellipse though the column 7 
lines 5-60 by the calculation of each element shown in Figure 6. 

With regards to claim 38, Leis discloses extracting a marker region from an original 
image as mentioned above. Maggioni discloses the apparatus that generates a reduced image of 
the original image extracting a candidate region that is considered to marker from the reduced 
image calculating the area of the candidate region within the original image and then extracts and 
recognizes a region corresponding to the marker within the original image at column 9 lines 30- 
39. Note, the original image is the image plane and the reduced image is the search area in 
Maggioni' s disclosure. 

Regarding claim 39, Leis discloses a marker having unique geometric characteristics (the 
markers 14a-14d having the shape of circle, Fig. 1). Maggioni discloses an apparatus that 
extracts a marker having no unique geometric characteristic (sometime in the shape of circle, 
sometimes in the shape of ellipse, column 6 lines 65-67 and column 7 lines 1-2). Note, 
Maggioni discloses the 3D position and orientation data when the marker is the shape of circle 
and also when the marker is the shape of ellipse. By utilizing the 3D positional relationship of 
the first marker, and update the 3D position and orientation parameter of the object with respect 
to the image apparatus, the non-unique marker has been obtained at column 8 lines 10-30. 



, Application/Control Number: 09/498,963 Page 9 

Art Unit: 2621 

With regards to claim 42, Leis and Maggioni both discloses markers. Leis does not 
specify the shape of the markers, hence, it would be obvious to have the markers in any shape, 
including a polygon. 

10. Claim 28 is rejected under 35 U.S.C. 103(a) as being unpatentable over Leis in 
combination with Batterman et al. (USPN 5,832,139). 

Regards to claim 28, Leis discloses markers having a pattern code that illuminates 
different energy (column 4 lines 50-54). Leis does not expressly teach the pattern code of 
markers in detail. Batterman discloses identification marks, which are disposed of coded 
patterns at column 6 lines 17-27. One of ordinary skill in the art would have been motivated to 
have different coded patterns markers to distinguish between patterns to find the six degrees of 
freedom (an object's position and orientation) as suggested by Batterman. 

1 1 . Claim 30 is rejected under 35 U.S.C. 103(a) as being unpatentable over Leis in 
combination with Saito (USPN 6,013,308). 

Regards to claim 30, Leis discloses a method finding the position and orientation of the 
3D object as mentioned above in claim 27, paragraph 7. Leis does not expressly teaches of using 
the apparatus on an another apparatus to measure the position and orientation relation ship of the 
two apparatus. Saito discloses having an image acquisition apparatus (robot 142 coupled to a 
camera 20 seen in Fig. 5) mounted on the other apparatus (machine 141) at column 6 lines 10-19 
and column 7 lines 8-17. One of ordinary skill in the art would have been motivated to have the 
image apparatus mounted on other apparatus to obtain the position and orientation of the object. 



f 



# 
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By moving the camera, which is coupled to the robot, it is easier to identify and remove the 
defects on the vehicle as described at column 6 lines 63-67 in Saito's. Note that Saito's 
invention is to remove the defects on the object. Identifying the defects is necessary part in 
removing the defects. 

12. Claim 32 is rejected under 35 U.S.C. 103(a) as being unpatentable over Leis and 
Maggioni as applied to claims 27 and 29 above, and further in view of Roch et al. (USPN 
5,138,667). 

Regarding claim 32, Leis discloses 3D position and orientation of the 3D object. Leis 
does not expressly disclose estimating error variance of the relative distance. Roch et al. (Roch) 
discloses an estimate error variance, which is calculated for the estimated relative distance at 
column 3 lines 40-48. One of ordinary skill in the art would have motivated to use the error 
estimation of Roch in combination with Leis position and orientation findings to eliminate any 
error in obtaining the correct solution for the position and orientation of the 3D object. 

13. Claims 35-36 are rejected under 35 U.S.C. 103(a) as being unpatentable over Leis and 
Batterman as applied to claim 28 above, and further in view of Ritter et al. (USPN6,363,169). 

With regards to claims 35-36, Leis discloses the markers as a fire marker (LEDs) where 
the markers have code pattern of the amount of energy focused on to the sensors. Leis does not 
expressly teach the markers having the patterns of different colors. Ritter et al. (Ritter) discloses 
circular marker utilizing as each marker, and small patterns of different colors are used as a code 
within the marker at column 18 lines 30-36. It is also clearly seen from the Figures 28 and 29 by 
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the indication of R, G, and B. At the time the invention was made, it would have been obvious to 
a person of ordinary skill in the art to use the different color for the patterns rather than the fire 
markers just having only one type of color. 

14. Claim 37 rejected under 35 U.S.C. 103(a) as being unpatentable over Leis in view of 
Maggioni as applied to claim 33 above, and further in view of Grimson et al. (USPN 5,531,520). 

Regarding to claim 37, Leis and Maggioni both discloses apparatus that obtains the 3D 
position and orientation of the object by having marks on the object. Neither Leis nor Maggioni 
discloses obtaining this information by adding direction information of the markers. Grimson et 
al. Discloses adding the direction information of the points (markers) at column 8 lines 40-48. 
One of ordinary skill in the art would have been motivated to find the position and orientation by 
adding the direction information rather than the method used in Leis and Maggioni as a preferred 
choice. 

15. Claim 41 is rejected under 35 U.S.C. 103(a) as being unpatentable over Leis in view of 
Pettersen et al. (USPN 5,440,392). 

With regards to claim 41, Leis discloses an apparatus to measure the position and 
orientation of the object. Leis does not expressly discloses the object being a sensor probe. 
Pettersen et al. discloses a sensor probe (element 14, Fig. 4) on which the markers are mounted 
on the plane at column 5 lines 54-62. Note that the light sources are considered as a marker (or a 
form of identification), which are sensed by a sensor 13 shown in Fig. 4. One of ordinary skill in 
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the art would have been motivated to use the sensor probe as an object to be identified in Leis to 
measure a position and an object coordinates. 



Allowable Subject Matter 
16. Claim 39 would be allowable if rewritten to overcome the rejection(s) under 35 
U.S.C. 1 12, second paragraph, set forth in this Office action and to include all of the limitations 
of the base claim and any intervening claims. 



Conclusion 

17. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

USPN 5921992 - Method and system for frameless tool calibration 
USPN 6304680 - High resolution, high accuracy process monitoring system 
USPN 6324296 - Distributed-processing motion tracking system for tracking 
individually modulated light points 

USPN 5905860 - X-Ray computed tomography system for detecting thing objects 
USPN 6144761 - Photogrammetric analytical measurement system 
USPN 6175647 - Method and system for 3D spatial position detection of surface points 
USPN 6222937 - Method and system for tracking vantage points from which pictures of 
an object have been taken 



• 
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Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Shefali d Patel whose telephone number is 703-306-4182. The 
examiner can normally be reached on M-F; 8:30am - 5:00pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Leo Boudreau can be reached on 703-305-4706. The fax phone numbers for the 
organization where this application or proceeding is assigned are 703-872-9314 for regular 
communications and 703-872-9314 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is 703-306-0377. 



Shefali Patel 
February 13,2003 




